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ABSTRACT 

This document contains 4 issues (spring 2002-winter 2002- 
2003) of "WCER Highlights," a publication of the Wisconsin Center for 
Education Research (WCER). Topics covered by the articles include: new tools 
for educational data management; the practice of science; the FAST (Families 
and Schools Together) program; South African students meet U.S. peers; 
testing accommodations; better teacher evaluation practices; services 
provided by Comprehensive Regional Assistance Centers; educational equity for. 
deaf and hearing students; 30 years of mathematics education research; 
measuring effects of systemic initiatives; new NSF (Nation Science 
Foundation) research centers at WCER; a home-grown professional development 
program succeeds; measuring the content of instruction; "academic enablers" 
and student success; putting case study results into context; and developing 
algebraic reasoning in the elementary school. (MES) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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New tools for educational data management 
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Transana software helps support researchers studying teacher-student interactions to build histories 
of student learning. 



V ideo data is an increasingly important resource for education researchers. High- 
profile studies such as the Third International Mathematics and Science 
Study-Repeat (TIMSS-R) showcase digital video of classroom practices as a cen- 
tral element of their research. 

Designers of new curricula and professional development materials are creating 
high-impact multimedia content that is based on rich studies of classroom practices. 

Video data can be more fully utilized when archived in digital format, well cata- 
logued, and readily accessible. WCER's Christopher Thorn and colleagues have devel- 
oped tools to help educators and researchers more effectively use digitized video data 
and other forms of related evidence. Thorn's project. Digital Insight, supports researchers 
as they acquire, manage, analyze, and disseminate video-based historical accounts of 
learning and development.' 
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FROM THE DIRECTOR 

Networking 

Building networks is an important part of life and 
learning. In this issue of Hi^hli^hts you'll read about 
a collaborative program in science and mathemat- 
ics education in which South African doctoral stu- 
dents network with peers in U.S. universities. The 
doctoral students report their experiences have 
helped them complete important research tasks, to 
think about the implications of their research, and 
to make breakthroughs in significant problems they had previously 
been unable to solve. 

To become scientifically literate, students need a knowledge of sci- 
entific practice as well as science content. WCER researchers are work- 
ing with ninth graders and their teachers to learn about science as it is 
actually practiced. This perspective marks a shift away from the view of 
science as a largely descriptive enterprise and toward the view that 
explanation is a central goal of scientific work. 

A program called Families and Schools Together (FAST) is enhanc- 
ing children's academic and social performance by strengthening net- 
works among parents, schools, and the community. Created in Madi- 
son, the program is now in practice in schools nationwide and in 
foreign countries. Students who have participated in the FAST program 
experience fewer behavioral problems and show improved academic 
performance. 

And to consider the technological side of networking: Designers of 
new curricula and professional development materials are creating more 
high-impact multimedia presentations. These materials can be better 
used when they are archived in digital format and stored on servers, 
well catalogued and readily accessible. Our cover story explains tools 
that help educators and researchers more effectively acquire, manage, 
analyze, and disseminate video-based historical accounts of learning 
and development. 

For more information about these stories and other research, visit 
our Web site at www.wcer.wisc.edu. 



Analyzing and annotating analog video is time 
consuming. But new video management software 
being developed by the Digital Insight team supports 
participants throughout the research process. 
Transana, for example, allows researchers to study 
much more data in the same time it would take to 
analyze a handful of data sets of analog video. The 
Digital Insight research environment also allows for 
more sophisticated analysis and for sharing results 
with researchers at a distance. 

The Digital Insight project is also designed to 
help researchers bring more forms of data to bear 
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on a problem and to make new arguments. It 
addresses the challenges researchers face in 

► video management, 

► human subject protection, 

► data management, and 

► use of archived multimedia data for secondary 
analysis and teaching. 

Researchers can extract event-based data — such as 
an instance of teacher-student interaction — from 
video observations. These events can then be cor- 
related with other kinds of data, such as field 
notes, images of student work, lesson plans, and 
assessment data. The goal is to build rich histories 
of student learning and development. 

The first product of the Digital Insight 
team is a video transcription and analysis 
tool called Transana, now available at 
http://www.transana.org/. Transana was originally 
designed by Thorn's colleague Chris Fassnacht to 
support his own dissertation research. 

Transana provides a way to view video, create a 
transcript, and link portions of the transcript to 
frames in the video. It facilitates identifying and 
organizing analytically interesting portions of 
videos and attaching keywords to those video 
clips. It also features database and file manipulation 
tools that help researchers organize and store large 
collections of digitized video. 

More specifically, Transana allows users to 

► import and view MPEG-1 videos,- 

► create transcripts of the videos,- 

► navigate through videos using several different 
mechanisms, including a graphic of audio 
soundtracks,- 

► link frames in the video to positions in a tran- 
script by embedding video time codes in the 
transcript,- 

► automatically highlight the relevant portion of 
the transcript while the video plays,- 

► select analytically interesting portions of the 
video, which can be grouped, - 

► define keywords and apply them to portions of 
video,- 

► search for instances of keywords and see the 
video clips to which they have been applied, - 
and 

► transfer video files to and from a storage 
resource broker (SRB), a computer system with 
massive storage capacity. 

continued on pa^e i 




The practice of science 

B y the time they get to high school, students 
have had at least 9 years of learning to play 
the “game" of school. They know what it 
takes to win: Just give the right answer. In the field 
of science, this approach places undue emphasis on 
the end products of scientific inquiry, depriving 
students of the opportunity to learn about the prac- 
tices through which scientific theories and explana- 
tions are constructed. 

Fortunately, instruction can be reorganized to 
dramatically change the nature of the school game. 
In new instructional contexts, students participate 
in making and assessing knowledge claims. They 
learn to recognize these activities as necessary to 
achieving understanding in science. 

Jennifer Cartier, Cynthia Passmore, and James 
Stewart argue that a knowledge of scientific prac- 
tice is an important part of becoming scientifically 
literate. Cartier, Passmore, and Stewart are part of 
a team of university researchers and local teachers 
known as MUSE (Modeling for Understanding in 
Science Education), headquartered at WCER's 
National Center for Improving Student Learning 
and Achievement in Mathematics and Science 
(NCISLA).* They have articulated a view of class- 
room understanding that is consistent with science 
as it is actually practiced. 

The practice franteivork designed by MUSE (see 
illustration) recognizes essential elements of 
inquiry, yet is general enough to be useful in class- 
rooms. The framework shows the central role of 
models in asking questions, recognizing data pat- 
terns, constructing explanations, and providing the 
criteria forjudging knowledge claims. 

Cartier and colleagues say this framework 
reflects a recent shift away from the view of sci- 
ence as a largely descriptive enterprise. The cur- 
rent view holds that explanation is a central, if not the 
central, goal of scientific endeavor. "Scientific 
inquiry is fundamentally about reducing the world 
to order," Passmore says. “Those reductions take 
the form of explanations." Inquiry in science is of 
primary importance. In fact, the MUSE team has 
identified the ability to participate in incluiry as an essen- 
tial component of understanding in science. 

Models in scientific practice 

Most often explanations involve the intersection of 
some causal model, or models, with data from the 
natural world. A scientific explanation is a careful 
mapping of a model to data. For example, an expla- 
nation for why we see phases of the moon 
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describes the movement of the Moon relative to 
the Earth and Sun (a model of celestial motion). 
This model results in the predictable data pattern 
of phase changes throughout the month. 

In some cases, models themselves become the 
objects of inquiry. When a new question is asked — 
for example, does light always behave as a wave? 
does genetic drift account for evolutionary changes 
in some species? — scientists assess existing models 
to see if they can address the question. If they can- 
not, scientists revise the existing models or develop 
new ones. 

Engaging students in exploring phenomena 
(such as the phases of the Moon) and developing 
or invoking models to account for those phenom- 
ena is a powerful way to get students started in 
classroom inquiry. As students explore specific 
phenomena, they participate in practice by devel- 
oping explanations and beginning to formulate 
their own questions. Students' questions are often 
quite sophisticated. For example, one student par- 
ticipant in MUSE research learned about the 
phases of the Moon by exploring local data. Then 
the student asked whether the same pattern would 
be visible from the southern hemisphere. This 
prompted him to gather information, print and 
electronic, to answer the question. In this case, the 
student was probing the fruitfulness of his model. 

continued on page 8 
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FAST strengthens families 



R ecent school violence has shattered the pub- 
lics perception that their children are safe. 
The roots of violence can often be traced 
to the family and the neighborhood. Parents are 
often too busy to spend time with their children, 
and neighbors do not know one another. Stretched 
thin by the demands of work and family and strug- 
gling to survive economically, parents find less and 
less time to socialize with other parents. The 
recent book Bowling Alone by Robert D. Putnam 
(Simon & Schuster, 2000) documents the deteriora- 
tion of friendships and other reciprocal relation- 
ships — social capital — in U.S. communities. People 
in schools and communities risk health and safety 
without the protection of networks of trusting rela- 
tionships. W^ithout social support, parents cannot 
provide enough support for their children as they 
face developmental stresses. Without caring par- 
ents, youth move toward dark futures of school 
failure, delinquency, drug and alcohol abuse, and 
even violence. 

To prevent these root causes from taking hold, 
WCER Senior Scientist Lynn McDonald initiated a 
school and community program in 1988 called 
Families and Schools Together (FAST). McDonald 



developed the FAST program to enhance children's 
academic and social performance. FAST programs 
build relationships among parents, schools, and the 
community. Multiple families from a school gather 
once a week for 8 weeks to share a meal and par- 
ticipate in a variety of structured social activities, 
including music, drawing, family games, children's 
sports, and opportunities for parents to interact in 
activities that apply theory and research. Students 
who have participated in the FAST program expe- 
rience fewer behavioral problems and show 
improved academic performance. 

To prevent violence and delinquency, McDon- 
ald says, it's important to work with younger chil- 
dren and their family networks. Early in children's 
lives, protective factors can make a big difference 
in development. Children in kindergarten and first 
grade are at an ideal age, and schools are an ideal 
location for intervention, with families having a 
central role. "All kindergarten children should have 
the chance to go to FAST," McDonald says. 

This past winter saw the completion of four 
multiyear research studies that used randomized 
trials to assess the impact of FAST. The studies 
evaluated the multicultural FAST program in 9 




New Orleans schools, 10 inner-city Milwaukee 
schools, 3 rural Wisconsin schools serving Native 
Americans, and 9 Madison schools. The results of 
these studies are consistent: More than one year 
after FAST, teachers and parents report FAST chil- 
dren show a statistically significant decrease in 
mental health indicators of risk for negative out- 
comes when compared with control group chil- 
dren. In some instances, the control group children 
showed significant increases in behavioral prob- 
lems over time, compared with the FAST children. 

If a family attends one FAST meeting, there is 
an 80% chance the family will graduate from the 
full 8-week program. This high retention rate is 
unusual, especially for stressed, low-income fami- 
lies. Two years after graduating, 86% of parent par- 
ticipants report that they are still seeing friends 
they made through FAST. Parents become friends 
and support one another over time. Some become 
community leaders. Using schools as community- 
based structures helps create outreach to families 
and becomes a viable approach for national repli- 
cation. 

The growth of FAST 

At WCER, McDonald directs the national dissemi- 
nation of FAST The program has been dissemi- 
nated to more than 600 communities in 38 states, 4 
Indian nations, and 5 other countries. A recent 
$1.9 million grant from the Bureau of Justice Assis- 
tance, U.S. Department of Justice, has enabled the 
launch of FAST as a major initiative, ensuring the 
quality of replicating this evidence-based model. 

In June, FAST officially became a Model Pro- 
gram for the Center for Substance Abuse Preven- 
tion (CSAP) at a ceremony at the National Press 
Club in Washington, D.C. Wisconsin was Just 
awarded a federal State Incentive Grant ($3 million 
annually for 3 years) to disseminate CSAP Model 
Programs across Wisconsin. The Wisconsin 
Department of FHealth and Family Services has 
been designated to administer the grant. 

Recognizing the program's success, U.S. 
F^ealth and FHuman Services Secretary Tommy G. 
Thompson said: 

"Communities across the country should insist 
upon and work toward excellence in helping chil- 
dren to succeed at school and at home, reduce 
drug and alcohol abuse, and reduce stress and 
social isolation. Families and Schools Together 
(FAST) and other programs supported by the fed- 
eral government have shown that prevention is 
possible and models of excellence are available." 

The state of Wisconsin allocated FAST $1 mil- 
lion annually from 1990 to 2000, and in 1994 then 

o 




Governor Tommy Thompson was recognized by 
FHarvard University and the Ford Foundation's 
Innovations in American Government awards com- 
petition for that statewide FAST initiative. Now 
four other states have statewide FAST initiatives. 
With the support of Senator F^erb Kohl (D-Wis.), 
FAST has received national recognition and federal 
research and development support since 1991. 

Bipartisan political support and funding from 
the United Way, foundations, local and state gov- 
ernments, and the federal government have helped 
FAST grow over time. In September 2000, 
McDonald was awarded a grant for work in 
Moscow by the National Institute on Drug Abuse, 
U.S. National Institutes of FHealth, through the 
U.S. -Russia Competitive Program. Eleven FAST 
families graduated in Moscow in December 2001, 
and McDonald visited there to conduct FAST 
training this spring. She also received a grant for 
adapting FAST to work with Hmong immigrant 
students and their families in Madison, Wisconsin, 
in conjunction with the United Refugee Services,- 
this grant was awarded by the Center for Mental 
F^ealth Services, U.S. Department of Health and 
Human Services. 

In addition to her duties as WCER Senior Sci- 
entist, McDonald is a part-time faculty member at 
Madison's Edgewood College in the Graduate Pro- 
gram in Family Therapy, and is president of the 
board of directors of the nonprofit FAST Interna- 
tional, which maintains quality assurance for the 
dissemination of FAST with certified trainers. 

For more information, contact McDonald at 
mrmcdona@facstaff.wisc.edu or visit the FAST 
Web site at www.wcer.wisc.edu/FAST. 



Family games and family 
meals at the school 
help build networks of 
parents. 
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South African students meet U.S. peers 




Students from South Africa attended a seminar with peers from Howard University in Washington, D.C. 
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S outh African science and mathematics educa- 
tion doctoral students studying at U.S. uni- 
versities in 2001 report that their U.S. work 
has helped them make progress in their research. 

A collaborative program was begun in 1997 to 
develop links between South African and U.S. sci- 
ence and mathematics education researchers on 
topics of mutual interest. One aspect of the pro- 
gram was the support of doctoral student educa- 
tion. UW-Madison Education Professor Peter 
Hewson oversees the U.S part of the program, 
which is jointly funded by the U.S. National Sci- 
ence Foundation and the South African National 
Research Foundation. 

Last year, 10 doctoral students who were regis- 
tered at South African universities spent 2 to 4 
months at selected universities in the U.S. They 
report that their experiences helped them progress 
in their graduate work in three ways: time for 
intensive study, access to more resources, and 
opportunities to share their work and receive feed- 
back from their peers. 



More time for thought 

The doctoral student program in science and 
mathematics education was conceived to serve sev- 
eral purposes. First, since South African students 
typically do their doctoral work part-time, the visit 
to the U.S. aimed to give them an opportunity to 
focus exclusively on their doctoral research. The 
idea was that intensive study of this nature would 
permit them to make significant progress in both 
conceptualizing and analyzing data for their disser- 
tations. Second, the research capacity of their host 
institutions in the U.S. provided them with access 
to significant resources, both human and material, 
for furthering their research. The program 
matched students with faculty mentors who shared 
their research interests and also gave them access 
to other faculty members, graduate students, 
courses, library holdings, and the like. Third, the 
South African students presented their work to 
audiences with interests and perspectives different 
from their own, which helped them clarify their 
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own research and provided them with confidence 
that their work represented a contribution to the 
field. Additionally, their participation in the acade- 
mic life of their host institutions and departments 
helped to enrich and broaden the dialogue and 
perspectives of their peers. 

The South African students took advantage of 
their host institutions' material resources, such as 
the journals, books, and online materials available 
through these institutions' libraries. Students 
reported that these resources helped them broaden 
their literature review, learn new methods of data 
analysis, and clarify issues in their research. 

Students also appreciated the opportunity to 
attend lectures, courses, seminars, workshops, and 
conferences. Students attended courses on a vari- 
ety of topics, such as science and mathematics edu- 
cation, discourse analysis, research methods in 
education, teaching for understanding, professional 
development, and the history of mathematics. 

Students received critiques of their work, both 
individually from faculty members and collectively 
from other graduate students. This feedback 
helped them clarify their thinking and sharpen 
their research focus, while also providing support, 
motivation, and encouragement to move forward 
with their doctoral work. Students reported that 
the feedback they received was especially affirm- 
ing when it indicated that the work they were 
doing was of interest, importance, and relevance to 
others in their fields of study. 

The program also offered the South African 
students the opportunity to participate in a day- 
long seminar with graduate students from Howard 
University, a historically black university in Wash- 
ington, D.C. The purpose of this facet of the pro- 



gram was to enable the students to explore the 
common ground between Howard's unique history 
in the U.S. and its research focus on oppression, 
on the one hand, and the South African students' 
research topics, on the other. Students affirmed the 
value of meeting students at Howard University 
and presenting their dissertation work in the day- 
long seminar. In particular, they appreciated the 
opportunity to discover the commonality of educa- 
tional issues addressed and the diversity of the 
approaches adopted in studying them. 

The next step 

So what happens to the students in this program 
when they return to South Africa? What factors 
will influence their ability to capitalize on their 
experience? According to Hewson, the hope is 
that the students have returned with an enhanced 
understanding of their research projects, a broad- 
ened set of professional contacts, and the personal 
motivation and initiative to complete their doctor- 
ates and move on in the development of their 
research expertise. 

At the same time, it must be recognized that, for 
the most part, the students will have returned to 
work environments where research capacity remains 
limited. Ongoing responsibilities in students' home 
institutions could limit their future research contri- 
butions. Thus, an important factor in their future 
development as researchers will be the availability 
of support, whether provided by their own institu- 
tions or by public or private granting agencies. 

For more information, contact Peter Hewson at 
pwhewson@wisc.edu. 
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Data management 

continued from page 2 

One way in which Digital Insight overlaps with the 
interests of related scholarly communities is 
Thorn's work with the TalkBank Project.^ Funded 
by NSF, TalkBank aims to create a distributed, 
Web-based data archiving system for transcribed 
video and audio data on communicative interac- 
tions. Thorn and his technical development team 
are working to develop model video annotation 
tools that support collection building and cross- 
disciplinary analysis. 

TalkBank's goal is to foster fundamental 
research in the study of human and animal commu- 
nication. Thorn's colleagues are developing stan- 
dards and electronic tools for creating, searching. 



and publishing video data via networked comput- 
ers. Six disciplinary groups make up the project's 
initial focus: Animal Communication, Classroom 
Discourse, Linguistic Exploration, Gesture and 
Sign, Text and Discourse, and Technical Develop- 
ment. Other disciplines will participate as the pro- 
ject grows. Digital Insight team members are active 
participants in several of these areas. 

For more information, contact Thorn at 
cthorn@wcer.wisc.edu or visit the project Web site 
www.talkbank.org. 

1 . Digital Insight is funded by the National Science Foundation 
through both the National Partnership for Advanced Computa- 
tional Infrastructure at the University of California-San Diego 
and the TalkBank project at Carnegie Mellon University. 

2. TalkBank Is funded by the National Science Foundation and 
Carnegie Mellon University. 
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Practice of science 

continued from page 3 

The MUSE team designed and 
implemented an introductory unit for 
ninth graders at a local, suburban Mid- 
western high school. The teachers used 
the unit to set the stage for the whole 
school's science sequence. One of the 
more complicated phenomena the stu- 
dents attempted to explain was that of 
the seasons. They identified several sea- 
sonal data patterns including the mid- 
day angular height of the Sun, average 
temperature, maximum shadow length, 
and average day length. The students 
found that all these patterns depended 
on both time of year and global loca- 
tion. The students were able to make 
sense of this complicated set of data 
with their teachers' help. The MUSE 
practice framework helped teachers 
redirect the students when necessary 
and focus their attention on creating 
explanations for patterns in nature, rather 
than simply attempting to offer the 
"right answer." 
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The MUSE view of scientific prac- 
tice as it occurs in classrooms may have 
important implications for the reform 
efforts under way in the U.S. If the goal 
of "understanding for all" is to be 
achieved, science educators must recog- 
nize that understanding in science 
develops through practice. Educators 
must design classrooms where realistic 
practice can happen. As a community, 
educators can go beyond a simple call 
for inquiry in science classrooms to a 
clear vision that can guide curriculum 
and professional development. 

For more information, contact Cartier 
at jcartier@facstaff.wisc.edu or Passmore 
at cmpassmo@students.wisc.edu, 
or visit the MUSE Web site at 
www.wcer.wisc.edu/ncisla/muse. 

1. Research is funded by the National Science 
Foundation and the Office of Educational 
Research and Improvement, U.S. Department of 
Education. 

Material in this article was originally delivered as a 
paper at the sixth annual conference of the Inter- 
national History, Philosophy and Science Teach- 
ing Group, November 2001. 
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Some closer looks at testing accommodations 




Four recent studies shed some light on the effects of accommodations on students' test scores. 



R ecent education reform efforts require high standards for all students. The inclu- 
sion of students with disabilities in statewide assessment systems is now required 
by law and considered to represent a key aspect of good testing practices. From 
social and accountability perspectives, this inclusion of students with disabilities is 
highly valued, especially when their test scores are known to be valid. Unfortunately, 
many students with disabilities receive testing accommodations of unknown validity. 

Including students with disabilities in assessment is important for two reasons: 

1 . it is critical to improving the quality of the educational opportunities available to 
these students, and 

2. it provides meaningful and useful information about students' performance to the 
schools and communities that educate them. 

Inclusion raises important questions, however. FHow appropriate are common perfor- 
mance standards for students with disabilities? What accommodations should be used? 
What are the effects of testing accommodations on the validity of assessment? FHow 
should scores be reported when accommodations have been used? 
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FROM THE DIRECTOR 

The roles of standards 

Standards continue to play an important role in 
K-12 education. In this issue of Highlights, you will 
read how standards are being used in the educa- 
tion of students with and without disabilities and 
in teacher evaluation. 

In a dissertation study, graduate student 
Stephanie Cawthon investigated inclusion poli- 
cies, such as those mandated by the Individuals 
with Disabilities Education Act (IDEA), that require educators to evalu- 
ate deaf students' participation in curricular content and assessment. 
Cawthon's results suggest that inclusive placement in the context of 
standards-based reform appears to result in equitable exposure to curric- 
ular content — that is, the opportunity to learn. 

In a related article, Stephen Elliott, Tom Kratochwill, and their 
graduate students report on the effects of accommodations on test 
scores of students with and without disabilities across several studies. 
Their general conclusion is that the accommodations investigated are 
not resulting in valid information about student learning. 

In another dissertation study, Steven Kimball studied how three 
school districts applied new standards for teaching — specifically, the 
model contained in Enhancing Professional Practice: A Frametvorkjor Teaching. 
Kimball concludes that new teacher evaluation systems are having posi- 
tive effects on teaching practice. Nevertheless some problems remain. 
Principal among them is the need for the evaluator to have subject-mat- 
ter competence. 

For more information, visit the WCER Web site at 
www.wcer.wisc.edu. 

Andy Porter 
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Accommodations are usually recommended in 
packages, rather than individually. 
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Education leaders at state and district levels 
struggle with these issues as they work to create 
policies for testing students with disabilities. In the 
past, many students with disabilities have been 
excluded from large-scale achievement tests. Rea- 
sons for the exclusion of students with disabilities 
are varied, but the most common are confusion 
about the use of test accommodations, concern 
over causing undue stress from testing, and fear 
that district test scores will go down. 

Four recent experimental studies conducted by 
UW-Madison Education Professors Stephen N. 
Elliott and Thomas R. Kratochwill and their gradu- 
ate students, with funding from the U.S. Depart- 
ment of Education's Office of Special Education 
and Rehabilitative Services, shed some light on 
current questions about the effects of accommoda- 
tions on test scores of students with and without 
disabilities. 

Packaged accommodations 

Elliott, Kratochwill, and student Brian McKevitt 
conducted a study (2001) designed to 

1 , describe the nature of information on testing 
accommodations listed in students' individual- 
ized education programs (lEPs), 

2, document the testing accommodations educa- 
tors actually use when administering perfor- 
mance assessments to students, and 

3, examine the effects accommodations have on 
the test results of students with and without 
disabilities. 

The study , found that accommodations were 
recommended in packages rather than individually 
- for example, the instructor might offer a combi- 
nation of verbal encouragement, reading the direc- 
tions aloud, simplifying some language, and 
rereading subtask directions. These accommoda- 
tion packages were found to have moderate to 
large effects on performance assessment scores for 
most students with disabilities and for some stu- 
dents without disabilities. This increase in scores 
for students without disabilities raises questions 
about the validity of the accommodations. If 
changes in testing procedure affect students with- 
out disabilities in the same direction and degree as 
they affect students with disabilities, the changes 
are not truly acting as accommodations. 

Constructed-response questions 

Graduate student Aleta Gilbertson Schulte con- 
ducted a study supervised by Elliott and Kra- 
tochwill (2001) to determine whether accommoda- 
tions on standardized tests affect students with 
disabilities differently than students without dis- 
abilkies. The researchers predicted that accommo- 




dations would significantly improve the test scores 
of fourth-grade students with disabilities but not 
those of students without disabilities. 

In fact, both groups of students improved sig- 
nificantly in the accommodated condition as com- 
pared with the nonaccommodated condition. 
Although students with disabilities benefited more 
than students without disabilities from accommo- 
dations on multiple-choice questions, the two 
groups benefited equally from accommodations on 
constructed-response questions. 

The finding that both groups of students expe- 
rienced benefits from testing accommodations on 
constructed-response questions indicates that the 
changes in test procedure may have affected both 
construct-relevant and construct-irrelevant vari- 
ance. The interaction between accommodation 
group and question type could indicate that con- 
structed-response questions are more difficult for 
all students and that accommodations remove bar- 
riers to these questions that are not present in mul- 
tiple-choice questions. 

These findings reinforce the principle that 
research on testing accommodations must take an 
individual perspective, and that all students must 
take the tests in both accommodated and nonac- 
commodated conditions if researchers are to deter- 
mine whether accommodations provide valid infor- 
mation on student learning. 

Extended time for tasks 

In a dissertation study (2000), Ann Marquart exam- 
ined the use of an extended-time accommodation 
on student scores on an eighth-grade mathematics 
test. Marquart found that giving students extended 
time to complete the test (40 minutes, rather than 
20 minutes) did not have significant results for stu- 
dents with or without disabilities. 

However, in a survey Marquart administered to 
students and their parents and teachers, most stu- 
dents reported feeling more comfortable, more 
motivated, and less frustrated under the extended- 
time condition. They thought they performed bet- 
ter, reported the test seemed easier, and preferred 
taking the test under the extended-time condition. 
Most teachers, but few parents indicated that a 
score from an accommodated test is as valid as a 
score from the same test administered without 
accommodations. Many parents, but no teachers 
believed that the score from an accommodated test 
is less valid than the score from a nonaccommo- 
dated test, and some members from both groups 
were uncertain. Most parents and teachers believed 
that if accommodations are used during test admin- 
istration, those accommodations should be 
rpnnrfed along with the test results. 





Reading test accommodations 

Graduate student Brian McKevitt (pictured above) 
studied the effects of testing accommodations on 
the scores of eighth-grade students on standard- 
ized reading tests, on score validity, and on teacher 
and student attitudes about testing (2001). 

McKevitt found that the accommodations rec- 
ommended by teachers did not significantly affect 
the test scores of students with or without disabili- 
ties. However, a read-aloud accommodation, when 
used with accommodations recommended by 
teachers, did positively and significantly affect test 
scores for both groups of students. 

There was much individual variability in the 
accommodation effects. The accommodations 
raised the scores for 50% of all students with dis- 
abilities and 38% of all students without disabili- 
ties. Although the scores of both groups were 
higher with the read-aloud than without any 
accommodation, the read-aloud effect was not sig- 
nificant when the scores of groups receiving the 
read-aloud accommodation were compared with 
the scores of groups receiving only the teacher- 
recommended accommodations. 

Elliott notes that educators who have cooper- 
ated with him and his students have rarely requested 
a summary of research on the effective use of testing 
accommodations. Perhaps they recognize there is 
little research on this issue, he says, "If research is 
going to guide practice, researchers and test pub- 
lishers interested in seeing all students participate 
meaningfully in assessments will need to help front- 
line educators more," he says. They need to under- 
stand which testing accommodations are most likely 
to be valid and how they can go about making deci- 
sions about the validity of testing accommodations 
for individual students prior to testing. 

For more information, visit the testing 
accommodations research Web site at 
http://www.wcer.wisc.edu/testacc/. 




Elliott 




Kratochwill 
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Statements of two 
teachers about 
expectations and 
input were typical 
across the districts. 
One said, "[The 
evaluations] are 
more two-way, 1 
think. In the past, 
you just went in 
and sat down and 
they did the [form]. 
Now, Tm more of a 
participator in the 
evaluation process." 




Better teacher evaluation practices 



O ver the last century, states and school dis- 
tricts have attempted to structure teacher 
evaluation practices to promote teacher 
accountability and improvement in practice. How- 
ever, traditional teacher evaluation systems and 
repeated attempts at reform appear to have done 
little to enhance either accountability or practice. 

Teacher evaluation systems are sometimes criti- 
cized for lacking commitment at the district or 
school level, using criteria reflecting narrow con- 
ceptions of effective teaching, providing inade- 
quate feedback, and operating with perceived sub- 
jectivity. 

Standards-based teacher evaluation represents 
one approach to addressing these problems. Stan- 
dards aim to represent a common conception of 
instruction. Proponents of new standards-based 
teacher evaluation suggest that feedback and 
objectivity can be strengthened when an evalua- 
tion system uses comprehensive standards and 
rubrics, along with multiple data sources and 
authentic samples of teaching work, to generate 
discussion of instruction between teachers and 
evaluators. Accountability for teaching to new 
standards and instructional improvement may 
result from such systems. 

UW-Madison assistant researcher Steven Kim- 
ball studied the design and operation of three dis- 
trict evaluation systems that use new sets of stan- 
dards for teaching — specifically, the standards 
proposed by Charlotte Danielson in Enhancing Pro- 
fessional Practice-. A Framework for Teaching (Alexandria, 
VA: Association for Supervision and Curriculum 
Development, 1996). Danielson designed her 
teaching framework generically so that it could be 
applied to various subject areas and to teachers 
with varying levels of experience. 

The three school districts Kimball studied had 
made considerable investments in time and person- 
nel resources to design, field-test, and implement 
new teacher evaluation systems. The systems 
aimed to enhance instructional accountability and 
teachers' professional growth. All three sites chose 
Danielson's framework for teaching because it 
► is comprehensive and applies empirical and 
theoretical research, 

► has multiple levels of performance defined by 
specific rubrics, 

► represents a conception of teaching that can be 
agreed upon by teachers and evaluators, 

► emphasizes teacher reflection and self-directed 
growth, and 



► saves the extensive time and effort that would 
be needed to research and develop new unique 
standards and evaluation criteria. 

Most of the teachers interviewed by Kimball 
considered their new evaluation systems substan- 
tial improvements over the prior systems. In com- 
parison with the prior systems, the new systems 
established more structure, provided more oppor- 
tunities for teacher discussion, and drew on more 
data sources for evaluation decisions. Teachers and 
others underscored the need for ongoing evaluator 
training to promote quality and consistency in 
evaluations. 

liming, credibility, and utility 

On the surface, most teachers interviewed by Kim- 
ball appeared to have a positive view of the perfor- 
mance feedback they received. But when their 
comments were examined more closely, it became 
clear that they had expressed a variety of percep- 
tions about the feedback timing, credibility, and 
utility, which influenced their experiences. In par- 
ticular, the perceived credibility of evaluators' feed- 
back varied, according to the teachers interviewed. 
Evaluator credibility was influenced by the amount 
of time evaluators had invested in the evaluation 
process and the extent to which teachers and eval- 
uators had similar content expertise. Time con- 
straints may have inhibited opportunities for feed- 
back and evaluation dialogue in some cases. 

Despite the increased workload associated with 
the new evaluation system, teachers largely saw 
the system standards, procedures, and outcomes as 
fair. Some teachers did question inconsistencies in 
evaluation data gathering and evaluator decisions. 
Increased administrator burdens, combined with 
insufficient training in the new systems, may have 
contributed to this problem. 

The quality of many teachers' performance was 
confirmed, and they felt encouraged to continue 
what they were doing. Others changed some of 
their instructional strategies based on the feedback 
they received. Still others did not consider the 
feedback particularly useful but nonetheless altered 
aspects of their instruction after reflecting on their 
performance in relation to the evaluation stan- 
dards. It was rare, however, for teachers to report 
substantial changes in their instructional practice 
as a result of their evaluation experiences, and the 
large majority of teachers did not see the evalua- 





Teachers would benefit from having evaluators familiar with their content area. 



tion process as an incentive to seek out profes- 
sional development opportunities. 

Untenured teachers were more likely than 
more experienced teachers to report improvements 
in their instruction based on feedback received 
during their evaluations. Kimball offers some 
explanations for this finding: First, untenured 
teachers were often novices with little classroom 
teaching experience. Many new teachers struggle 
with the most basic (yet important) aspects of 
teaching: lesson organization, record keeping, and 
classroom interactions. Principals and other evalua- 
tors were often able to draw on their prior teach- 
ing experience and their work as instructional lead- 
ers to provide constructive feedback on these 
practices that novice teachers could put to immedi- 
ate use. 

Second, evaluators were required to spend 
more time with untenured teachers, observing 
more of their classroom sessions and conducting 
more pre- and postobservation conferences. The 
additional time evaluators devoted to assessing 
untenured teachers gave these teachers more 
opportunity for direct and ongoing feedback. 

Based on his study, Kimball concluded that 
each of the three districts could benefit from 
addressing three fundamental questions related to 
the implementation of standards-based teacher 
evaluation reforms: 



► Can other evaluators, besides the school-based 
administrators, provide effective evaluations? 

► How does one strike a balance between mak- 
ing valid decisions about performance, on the 
one hand, and minimizing evaluation burdens, 
on the other? 

► What is the appropriate amount and nature of 
evaluator training? 

In the districts studied, administrators who tra- 
ditionally conducted teacher evaluations (primarily 
principals) were chosen to implement the new 
evaluation systems. As a result, some teachers were 
evaluated by administrators with little or no expe- 
rience in the teachers' content area or grade level. 
This problem was most common at the high 
school level, given secondary teachers' greater spe- 
cialization in content fields. 

Evaluators' own content backgrounds affect 
their ability to make informed inferences about the 
quality of teachers' instruction on specific content. 
Evaluators' backgrounds also may limit their ability 
to provide deep feedback. One solution, leading to 
deeper analysis and more credible feedback on con- 
tent-related instruction, would be for evaluators to 
develop in-depth knowledge about teaching and 
learning in one subject. 

For more information, contact Kimball at 
skimball@education.wisc.edu, (608) 265-6201. 
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Comprehensive Centers offer silent service 




Cotherman 



F ifteen Comprehensive Regional Assistance 
Centers around the country provide technical 
assistance, professional development services, 
educational materials, and distance learning to 
educators in their regions. Most Comprehensive 
Centers' customers gave the centers high ratings 
for the accessibility, quality, and utility of their ser- 
vices. ^ The Midwest regional center, the Compre- 
hensive Center-Region VI (CC-Vl) serves six 
states and is based at WCER. Center Director 
Audrey Cotherman calls the assistance provided by 
CC-Vl a "silent service," involving practitioners 
thoroughly and deeply and enabling them to find 
their own solutions. 

Congress created the Comprehensive Regional 
Assistance Centers in 1995 to provide quality ser- 
vices to schools, districts, state education depart- 
ments, community members, and Bureau of Indian 
Affairs schools. 

In the last 6 months, WCERs CC-Vl staff have 
served 22,500 educators in North Dakota, Min- 
nesota, Wisconsin, South Dakota, Iowa, and 
Michigan. CC-Vl also coordinates its services with 
state departments of education to determine 
statewide needs. Principals and other administra- 
tors are involved by CC-Vl in developing ways to 
change school cultures and to support high perfor- 
mance by teachers and students. CC-Vl activities 
are funded by the U.S. Department of Educations 
Office of Elementary and Secondary Education. 




Teachers' classroom practice benefits from research-based training. 



Center mission 

Before the formation of the 15 comprehensive cen- 
ters, there were 46 assistance centers for different 
programs. Congress consolidated these centers 
into the 15 comprehensive centers so that practi- 
tioners could more easily find assistance. The cen- 
ters were designed to provide a comprehensive 
approach to school improvement by focusing on 
whole school reform, and to view the special needs 
of special populations within the context of high 
performance for all students. 

CC-Vl trainers work with educators at their 
schools over a sustained time to improve the 
teaching and learning of reading or math,- to train 
educators on how to collect and use data for deci- 
sion making,- and to provide assistance in forming 
professional learning communities. 

CC-VI trainers also publish and distribute 
research studies,- share effective ways to involve 
parents as teaching partners,- guide schools in 
working toward safe and drug-free environments, - 
and share research on best practices for teaching 
non-English-speaking students. 

Generally, CC-VI's training programs focus on 
seven research-based areas: reading, math, English 
as a second language, the role of the principal as 
instructional leader, student assessment, parental 
involvement, and smaller learning communities. 

CC-VI staff respect and use the knowledge 
practitioners already have. They design structures 
for educators to share their successes,- they 
acquaint educators with results of education 
research,- show teachers and principals how to 
implement the research in the classroom,- and pro- 
vide sustained rather than one-time or sporadic 
training, materials, and guidance. 

Summer institute 

Throughout the year, CC-Vl staff spend most of 
their time on the road, visiting educators in their 
home districts. Each summer, CC-Vl offers numer- 
ous institutes. This summer, educators participated 
in the Fifth Annual CGI National Institute on the 
UW-Madison campus. 

CGI (Cognitively Guided Instruction) is a 
problem-solving mathematics program, developed 
at WCER, for students in kindergarten through 
Grade 3. CCl has proven effective for boys and 
girls of diverse socioeconomic, racial, ethnic, and 
language backgrounds. Originally developed and 
tested in Madison and Madison area schools, this 
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Educational equity for deaf and hearing students 



S tudents with disabilities, even those in main- 
streamed classrooms, have historically 
received lower quality instruction and have 
often been excluded from the required curriculum. 

Many classrooms now do include students 
(including deaf students) who traditionally would 
have been placed in special classes. At the same 
time as deaf students are increasingly integrated 
into diverse educational settings, national and state 
agencies are calling for higher standards of student 
achievement. Inclusion policies, such as the Indi- 
viduals with Disabilities Education Act (IDEA), 
require educators to evaluate deaf students' partici- 
pation in curricular content and assessment. How- 
ever, it is not clear whether inclusive placement 
results in equitable exposure to curricular content — 
that is, equitable opportunity to learn (OTL). 

UW-Madison graduate student and WCER 
research assistant Stephanie Cawthon recently com- 
pleted a study of the access hearing-impaired stu- 
dents have to the standards-based reading curricu- 
lum. Her study adapted methods pioneered by 
WCER Director Andrew C. Porter and his col- 
leagues to study the "enacted curriculum" (i.e., what 
is taught and how). The measure of the enacted cur- 
riculum developed by Porter and his colleagues 
evaluates the relationship — or alignment — between 
content standards and classroom practices in core 
content areas. It provides information on content 
coverage in both time (number of classes) and depth 
(level of cognitive work asked of students). 

Cawthon investigated in particular how the 
Wisconsin state standards for reading are covered 
in classrooms with deaf and hearing students. 
Cawthon's study was unique in a couple of ways: 
First, by addressing standards for reading in the 
early elementary grades, it expanded the study of 
alignment to a new subject area. Second, it focused 
on a special education population and educational 
equity across classroom settings. 

Cawthon asked teachers to complete a reading 
curriculum survey describing their instruction dur- 
ing the spring 2001 semester. Teachers were asked 
to indicate how much time they allocated to spe- 
cific reading curriculum topics (i.e., plot, character, 
spelling, application of information from world 
events) and what types of learning goals they had 
for their students (e.g., to memorize, understand, 
apply, and analyze material). 

Teachers in deaf-only, mixed, and hearing-only 
classrooms reported equitable exposure to stan- 
dards-based curriculum. This is a promising result 
in the context of inclusive placement and stan- 
O based reform. The IDEA emphasizes the 

ERLC 





Standards-based analyses may not capture all 
differences in instructional experiences. 



need for a continuum of services based on the 
needs of deaf students. Cawthon's results indicate 
there is less need for concern over possible differ- 
ences in OTL standards in different educational 
settings for deaf students. 

However, notwithstanding Cawthon's finding 
of equitable OTL, it is still likely that teachers 
have different reading instruction strategies for dif- 
ferent students. In other words, a standards-based 
analysis may not capture what are real differences 
in the instructional experiences of different partici- 
pant groups. One way to graphically depict how 
instruction time is used is by constructing content 
maps. These figures are topographical maps that 
show relative "highs" and "lows" of instructional 
time, similar to the "mountains" and "valleys" over 
rough terrain. They demonstrate differences in 
how teachers implement standards-based curricula, 
and show levels of alignment with standards. 
Diversity in instruction from one teacher to the 
next is not only to be expected, but hoped for, 
given the range of communicative and educational 
needs of deaf and hearing students. Thus, differ- 
ences in how teachers implement standards is a rich 
area for further research. 

Cawthon's study has implications for the 
emerging field of alignment analysis in standards- 
based reform. Educators assume that higher levels 
of alignment to standards will result in higher lev- 
els of academic achievement on assessments tied to 
those standards. Yet achieving better results 

continued on page 8 
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Comprehensive centers 
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program has been replicated in Austin 
and San Antonio, Texas,- Los Angeles, 
California,- Dearborn and Lansing, 
Michigan, Milwaukee, Wisconsin,- 
Prince George's County, Maryland,- Bis- 
marck and Fargo, North Dakota,- Wash- 
ington, D.C.,- and numerous other sites. 
This year's institute was specia lly 
designed for teams composed of a 
trainer and two or more primary grade 
teachers. Over five days, participants 
learned how critical mathematical ideas 
develop in children and planned how to 
refocus instruction to build on children's 
natural mathematical abilities and to 
integrate the learning of skills and prob- 
lem solving. 

For more information, visit the CC-VI 
Web site at www.wcer.wisc.edu/CC-VI,- 
e-mail CC-VK^mail. wcer.wisc.edu,- or 
call 608-263-4220. 

•To view a copy of the report "Comprehen- 
sive Regional Assistance Centers Program: 
Final Report on the Evaluation" (2000) point 
your browser to: http://www.ed.gov/offices/ 
OUS/PES/ExecutiveSummary 1 1 .1 .OO.PSAcor 
recteddoc.doc 



Deaf and hearing 

continued Jrom page 7 

involves a substantial number of other 
variables — for example, school 

resources, class size, teacher characteris- 
tics, student characteristics, assessment 
validity, and professional development. 
The strength of the relationship 
between degree of alignment and stu- 
dent achievement thus is modified by 
these additional factors and needs to be 
verified by current and future research. 

Alignment between components of 
accountability measures is one proposed 
method of documenting the success of 
standards-based reform. 

Cawthon says that this area of 
research must also identify the educa- 
tional significance of alignment between 
standards, curriculum, and assessments. 
In the hearing-only, mixed, and deaf-only 
classrooms investigated in this study, 
teachers report their reading instruction 
to be moderately aligned to the stan- 
dards. Perhaps moderate alignment to 
standards produces "good enough" results 
in terms of student achievement. Studies 
of simultaneous alignment with standards 
and related assessments will be needed to 
clarify the educational significance of 
these findings. 

For more information, contact 
Cawthon at stephanie_cawthon@hot- 
mail.com. 
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30 years of mathematics education research 




Learning with understanding means more than being able to produce correct answers to routine problems. 



R esearchers and educators have learned a lot about the teaching and learning of 
mathematics in schools during the past 30+ years, much of it from the work of 
the emerging mathematics education research community. However, UW— Madi- 
son education professor emeritus Thomas A. Romberg says researchers and educators 
still have lots to learn about the teaching and learning of mathematics in the 'messy" 
social environment of school classrooms. 

For research to be productive and useful in any discipline, it must be conducted 
within a research community. Mathematics education is a relatively young academic 
field, and research on the teaching and learning of mathematics is even younger. A 
research community in mathematics education has only gradually begun to emerge in 
the past half century, and the process is still ongoing. 

Reviewing mathematics education research, Romberg identifies 12 findings that dis- 
tinguish what is known today from what was known in the 1960s. The first five come 
from research conducted by the National Center for Improving Student Learning and 
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FROM THE DIRECTOR 

WCER welcomes two new Centers 

Improving mathematics and science education has 
been at the core of WCER's work for decades. We 
continue that tradition with the introduction of 
two major new projects funded by the National 
Science Foundation. System-Wide Change for All 
Learners and Educators (SCALE) aims to reform 
math and science education, Grades pre-K 
through 12. It involves a UW-Madison partner- 
ship with the University of Pittsburgh and four school districts around 
the country. Another new project, the Center for the Integration of 
Research, Teaching and Learning (CIRTL), is part of a network of NSF 
Centers for Learning and Teaching. Together, these two centers address 
reform in grades kindergarten through college. They seek to develop a 
national science, engineering and mathematics faculty with teaching 
skills that will enable all college students to be scientifically literate. 

1 call my friend and colleague Tom Romberg "Mr. Math Education" 
because for decades he has made significant contributions to the 
nations mathematics policy. In this issue Tom reviews his more than 30 
years of research and discusses several things mathematics educators 
know today that they didn't know in the 1960s. 

Systemic reform offers the possibility of fundamental improvement 
in American education. A question that remains is, how precisely can 
one measure student achievement resulting from systemic reform 
efforts? WCER researcher Norman Webb and colleagues recently iden- 
tified three factors that can influence such measurement. 

Also in this issue, UW~Madison education Professor Richard 
Halverson shares a story that illustrates how a home-grown program in 
a Chicago urban school succeeded in increasing the achievement of stu- 
dents of color and low-income students. The school principal launched 
a voluntary monthly "breakfast club" at which teachers gather to discuss 
their practice. 

You're invited to spend time at our web site, www.wcer.wisc.edu. 
There you'll find research news beyond what we report in this newsletter. 

Andy Porter 




Achievement in Mathematics and Science on stu- 
dent learning with understanding, located at 
WCER. 

1. Educators have underestimated students' capa- 
bility to learn mathematics with 
understanding. Given the opportunity to 
explore a domain using a set of structured 
activities, all students can leant important mathematics 
with understanding. 

2. Learning the concepts and skills in a mathe- 
matical domain requires that students be 
engaged in a rich set of structured activities 
over time. Specifically, students need an oppor- 



tunity to investigate problems that encourage 
mathematization. By that is meant problems that 
are subject to measurement and quantification, 
that embody quantifiable change and variation, 
that involve specifiable uncertainty, and that 
involve our place in space and the spatial fea- 
tures of the world we inhabit and construct. In 
addition, problems should encourage the use of 
languages for expressing, communicating, rea- 
soning, computing, abstracting, generalizing, 
and formalizing. Such problems require sys- 
tematic forms of reasoning and argument to 
help establish the certainty, generality, consis- 
tency, and reliability of students' mathematical 
assertions. 

3. Learning with understanding involves more 
than being able to produce correct answers to 
routine problems. Mathematics should be 
viewed as a human activity that reflects the 
work of mathematicians- — finding out why 
given techniques work, inventing new tech- 
niques, and Justifying assertions. Learning with 
understanding occurs when it becomes the 
focus of instruction, when students are given 
time to discover relationships and learn to use 
their knowledge, and when they reflect about 
their thinking and express their ideas. Doing 
mathematics cannot be viewed as a mechanical 
performance or an activity that solely involves 
following predetermined rules. 

4. Modeling and argumentation are important aspects 
of mathematics instruction that foster learning 
with understanding. Modeling offers a way to 
represent phenomena in the world by means of 
a system of theoretically specified objects and 
relations. Modeling is critical in developing 
understanding in a domain. In classrooms, it is 
important to consider modeling as a cycle 
comprising model construction, exploration, 
and revision. Additionally, as students make 
conjectures, they need to learn to Justify them. 
Thus, argumentation and standards of evi- 
dence, with an emphasis on promoting stu- 
dents' skills for generalization in mathematics, 
are critical. 

5. Student learning should be seen as a product of 
situated involvement in a classroom culture. 
Learning with understanding is a product of 
interactions over time with teachers and other 
students in a classroom environment that 
encourages and values exploration of problem 
situations, modeling, and argumentation. The 
very nature of mathematics is defined commu- 
nally, making participation by all not only a 
fundamental civil right, but also a critical pre- 
requisite to the continued vitality of mathemat- 
ics in the nation. 





The very nature of mathematics is defined communally. 



Romberg also points to four general findings from 
research on teaching. The reform approach to 
teaching represents a substantial departure from 
most teachers' prior experience, established beliefs, 
and present practice. 

6. Teacher knowledge of student thinking is criti- 
cal. Teachers need to listen and hear what stu- 
dents are saying as they conjecture and build 
arguments. Teachers also need to Judge the 
quality of students' Justifications and explana- 
tions in examining student work. 

7. Teachers must understand the structure of 
mathematical domains. Knowledge of the net- 
work of relationships in a domain is critical 
when making decisions about student under- 
standing and the sequence of instruction. 

8. Rather than Just cover the content in a text- 
book, teachers need to base instruction on the 
needs of their students. This finding follows 
directly from the previous two. If teachers 
know the level of their students' thinking, and 
understand how it fits within the structure of 
the domain of interest, then they can design 
appropriate instruction. 

9. Professional development cannot be done well 
in isolation. Professionalism is the key to qual- 
ity classroom instruction, but it can only be 
achieved if teachers Join together to collabora- 
tively undertake professional assistance. 

Finally, one of the key aspects of teaching has 
always been monitoring students' progress. Mathe- 



matics teachers have traditionally done this by giv- 
ing curriculum-based quizzes and tests, scoring and 
counting the number of correct answers on each, 
and periodically summarizing student performance 
in a letter grade. Unfortunately, this standard 
approach to assessment is not consistent with what 
educators now know about how students learn 
mathematics. 

10. External tests have an effect on instruction in 
that teachers take classroom time to prepare 
students to take the test. However, such tests 
are not often well tailored to classroom instruc- 
tion, nor are the results useful for monitoring 
growth over time. 

11. Curriculum-based quizzes and tests tend to 
include items that are very similar to exercises 
in daily lessons and that reflect reproduction, 
definitions, or computations. Such instruments 
rarely contain items that require students to 
relate concepts or solve nonroutine problems. 

12. Most mathematics teachers are aware that they 
acquire considerable informal evidence about 
their students, yet they rarely use such evi- 
dence in judging student progress. In fact, cur- 
rent data show that most mathematics teachers 
could benefit from professional development 
designed to help them learn how to make good 
use of their informal assessments. 

Schools are social and political organizations that 
operate within a coherent set of traditions. Chang- 
ing such organizations involves understanding and 
dealing with the partisan political and ideological 
perspectives on schooling that permeate our society. 

Researchers and educators who hope to ground 
school procedures in the findings of research 
rather than in politics and ideology are likely to 
face either of two arguments: that the research is 
based on grossly unrealistic reductions of complex 
phenomena, or that it involves conflicts of value 
that cannot be resolved by evidence. In fact, 
Romberg believes it is naive to believe that 
research findings can curtail partisan prejudices 
about schooling. Nevertheless, research is increas- 
ingly providing insights, understanding, and new 
approaches that lead to instruction that more 
effectively promotes student achievement. 

For more information contact Romberg at 
tromberg@facstaff.wisc.edu. 

Funding Jor research conducted by the National Center Jor 
Improving Student Leantin^ and Achievetnent in Mathematics 
and Science (NCISLA) is provided by the U.S. Department 
of Educations Office of Educational Research and Improve- 
ment. 
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Thomas A. Romberg is 
Sears Roebuck Founda- 
tion-Bascom Professor of 
Education at the Univer- 
sity of Wisconsin— Madi- 
son. His research has ana- 
lyzed young children's 
learning of initial mathe- 
matical concepts, meth- 
ods of evaluating students 
and programs, and the 
integration of research on 
teaching, curriculum, and 
student thinking. 

Romberg was a leader 
in the development of the 
National Council of 
Teachers of Mathematics 
(NCTM) Curriculum and 
Evaluation Standards for 
School Mathematics 
(1989). He is former 
director of WCER's 
National Center for 
Improving Student Learn- 
ing and Achievement in 
Mathematics and Science. 
He offers these observa- 
tions looking back over a 
career spanning more 
than 30 years. 
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Measuring effects of systemic initiatives 




Do systemic initiatives improve student achievement? New measurement tools point toward answers. 




S ince public schools were first instituted in the 
United States, society's needs have changed. 
One way for education to keep up with a 
changing society is through piecemeal reform — 
that is, programs that offer improvement in spe- 
cific areas of student achievement. Systemic educa- 
tion reform, on the other hand, offers the 
possibility of fundamental improvement in Ameri- 
can education. 

Demonstrating the effects of systemic educa- 
tion reform is difficult, given its complexity. How- 
ever, a recent study by Norman Webb and WCER 
colleagues demonstrates analytic techniques that 
can be used to study the effects of systemic reform 
on growth in student learning over time. 

Webb and colleagues analyzed data from the 
Texas Assessment of Academic Skills (TAAS) for 
Grades 3 through 8, focusing on the years 1994 
through 2000. They then compared student 
achievement in Urban Systemic Initiative (USI)^ 
districts with achievement in other Texas dis- 
tricts. Webb's analysis produced evidence of 
improvement in student learning in USI districts as 
compared with other districts. Specifically, the 
team found that: 



► Students' TAAS scores in USI districts 
improved from 1994 to 2000 for all groups. 
Annual gain scores by Black and Latino stu- 
dents improved over time relative to those of 
White students. 

► Overall, student achievement scores in USI 
districts began lower but improved faster than 
those in non-USI districts. 

► Comparing USI and non-USI districts, one 
finds no difference in the rate at which gaps in 
achievement among students of different socio- 
cultural backgrounds are narrowing. 

Questions addressed 

Webb's multidisciplinary research team worked for 
over a year to develop an analytic framework for 
studying the degree to which systemic reform con- 
tributes to improved student achievement and 
other outcomes. 

For example, team member Dan Bolt examined 
changes in school mean scores on TAAS at a given 
grade level (e.g., Grade 5 in 1994, 1995, 1996, 
etc.). He believed this approach would more effec- 
tively control for teacher effects because the same 







teachers are more likely to teach the same grade in 
successive years. 

Meanwhile, Adam Gamoran used nearly all of 
the students in the database to estimate the growth 
intercepts and slopes. In this model, students with 
any two scores, even those whose scores are not 
for consecutive years, can be used to estimate the 
parameters. 

Robert Meyer examined students' performance 
in a grade by considering their achievement from 
the year before. His analyses included students 
who had test scores for two consecutive years. The 
advantage of this approach is that improved stu- 
dent performance can be measured more precisely 
than if only one school year is considered. 

All three approaches produced evidence that 
USl school districts had at least a small positive 
effect on student achievement. However, because 
of the lack of more specific information on USl 
activities, Webb cautions that the models could 
not definitively answer the question of whether the 
effects were directly related to USl participation. 

The team did provide information to NSF that 
will enable NSF, its education constituencies, and 
education researchers to address the following 
questions: 

► How can the data submitted to NSF by sys- 
temic initiatives be used to evaluate systemic 
reform? 

► How does the precision of analysis depend on 
the qualities of student assessment data? 

► What statistical models best fit the data linking 
systemic initiatives to student achievement? 

► What are the lessons learned about the kinds of 
databases and analyses that are most effective 
for evaluating and understanding systemic 
reform? 

Factors affecting precision 

Webb and colleagues identified three general fac- 
tors that can influence the precision of analyses of 
student achievement data in relation to systemic 
reform: 

1. The extent to which teachers, schools, and dis- 
tricts participated in the systemic initiative 
over time. Classifying schools by their degree 
of participation in systemic initiatives would 
allow comparing school performance and 
would provide more precise information. 

2. The types of students excluded from the test- 
ing and analyses,- and 

3. The standard error of measurement in the 
assessment instruments. 




Achievement scores began lower but improved faster 
in USl districts. 



Webb and colleagues hope that the analytic mod- 
els they developed for this study will be widely 
applicable to other studies of large-scale reform. In 
fact, the most important contribution of the study, 
according to Webb, is its potential to inform the 
design of other evaluations of large-scale reform 
efforts and thus to increase the likelihood that data 
will be available in the future to more effectively 
measure the impact of such interventions on stu- 
dent learning. 

Webb cautions that there is no one best model 
for analyzing the link between systemic initiatives 
and student achievement. Each model is based on 
specific assumptions made necessary by the limita- 
tions of available data or other constraints. 

The research conducted by Webb and col- 
leagues was supported by the National Science 
Foundation. For more information, contact Webb 
by e-mail at nlwebb@facstaff.wisc.edu or by tele- 
phone at (608) 263-4287. 

i. NSF launched the USl program in i994, applying lessons 
learned from its earlier State Systemic Initiative (SSI) program to the 
problems of inner-city school systems. The USl program was offered 
to major cities with the largest number of K—i2 students living in 
poverty. 
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WCER hosts new NSF research centers 
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T he National Science Foundation has 
awarded WCER $45 million over the next 5 
years for two new mathematics and science 
education research centers, System-Wide Change 
for All Learners and Educators (SCALE) and the 
Center for the Integration of Research, Teaching 
and Learning (CIRTL). 

SCALE aims to reform math and science edu- 
cation, Grades pre-K through 12. It involves a 
UW— Madison partnership with the University of 
Pittsburgh, the Los Angeles Unified School Dis- 
trict, Denver Public Schools, Providence Public 
Schools, and the Madison (Wis.) Metropolitan 
School District. SCALE is bringing together math- 
ematicians, scientists, social scientists, and educa- 
tion practitioners to improve the math and science 
achievement of all students at all grade levels in 
the four school districts. Students will be engaged 
in deep and authentic science and mathematics 
instructional experiences. Documentation of what 
works and information about how to construct 
such a partnership will be made available to policy- 
makers and university and school leaders. 

SCALE is directed by Terrence Millar, 
UW-Madison Professor of Mathematics and Asso- 
ciate Dean of the Physical Sciences, Graduate 
School. 

The new Center for the Integration of 
Research, Teaching and Learning (CIRTL) 
promises to transform the UW-Madison into a 
working laboratory for helping graduate students 
and faculty develop teaching skills that match their 
research skills. Part of a network of NSF Centers 
for Learning and Teaching, CIRTL seeks to 
develop "a national science, engineering and math- 
ematics faculty with teaching skills that will enable 
all college students to be scientifically literate, and 
that will promote a public better prepared to live 
in a high-tech world," says Robert Mathieu, a 
UW— Madison astronomy professor and the princi- 
pal investigator of the new initiative. 

CIRTL aims to help ensure that math and sci- 
ence are taught well not only to the select few 
undergraduates who go on to advanced degrees 
and careers in these fields, but also to students who 
will encounter only a minimum of science and 
math coursework. "All students should profit from 
improved instruction in undergraduate math and 
science, not just those pursuing a major," says 
Mathieu. "Students of color and women, for exam- 
ple, are less likely to take math and science courses 
as undergraduates. Even when they do, they are 
less likely to pursue further study in those disci- 
plines." 




CIRTL will aim for improved instruction in undergrad- 
uate math and science nationwide. 



CIRTL will research and implement ways to 
increase the effectiveness of teaching approaches 
for all undergraduate students and thereby enhance 
success in science courses for diverse audiences. 

CIRTL is a partnership of UW-Madison with 
Michigan State and Penn State Universities. 

These projects join WCER's other recently 
funded NSF Center for Learning and Teaching, 
Diversity in Mathematics Education (DiME). 
DiME aims to develop and enhance the instruc- 
tional workforce from kindergarten through gradu- 
ate school. The program consists of three interre- 
lated components: a doctoral/postdoctoral 
component; a teacher education component for 
teachers and instructional leaders,- and a compre- 
hensive research agenda. These components are 
integrated by a focus on the ideas of algebra and 
issues related to learners with diverse cultural, lan- 
guage, and cognitive backgrounds. DiME is 
directed by Walter Secada, UW-Madison profes- 
sor of curriculum and instruction. 

The centerpiece of DiME is its community of 
scholars, including the faculty, doctoral and mas- 
ter's degree students, participating teachers, and 
undergraduates who will engage in collective 
analysis of cases of mathematical learning and 
teaching using Web-based software. ■ 







Breakfast, anyone? 

Home-grown professional development program succeeds 

Richard Halverson works to make the wisdom of successful leadership practice available to interested 
practitioners. Schools are very often mandated to change without principled opportunities to learn from 
successful peers. His current research with associate Colleen Capper aims to create multimedia cases of 
leadership practice that 

► communicate the goals and resources required for leaders to engage in school change, and 

► tell the story of how practitioners weave resources together to form coherent practice in 
their schools. 

Practical wisdom is disclosed in the ways that school leaders set and solve problems in their schools. 
Halverson looks at locally-designed programs, which he refers to as artifacts, that provide a window into 
the practical wisdom of school leaders. In one case Halverson studied recently, one school's 
"breakfast club" showed how school leaders saw student achievement as a matter of improving the 
professional community in the school. Breakfast Club was one of the tools school leaders used to help 
develop and guide the emergent sense of professional community. 




Halverson 



T eaching does not begin and end in the class- 
room. A teachers experiences with other 
faculty members, with the school's leaders, 
and with its organizational structure all have a pro- 
found effect on the teacher's influence on students. 

UW-Madison education professor Richard 
Halverson and colleagues documented the prac- 
tices of public school principals from a variety of 
communities who succeeded in increasing the 
achievement of students of color and low-income 
students. They recently found a Chicago public 
elementary school that raised its student achieve- 
ment scores after the school principal launched a 
voluntary monthly "breakfast club" at which teach- 
ers could gather to discuss their practice. Halver- 
son was particularly interested in this innovative 
program as a model of distributed leadership — that is, 
leadership that emerges through the interaction of 
leaders and followers in the execution of both the 
everyday tasks of leadership (the micro tasks) and 
the school's overall instructional goals (the macro 
tasks). 

Adams School is a Chicago neighborhood ele- 
mentary school with about 1,200 students (largely 
African American) housed in two sites. Adams is 
widely recognized as a school with a well -articu- 
lated vision and a record of instructional change. 
Over the past 10 years, Adams has recorded 
demonstrable gains in student performance on 
high-stakes district and state assessment measures, 
and school leadership is given much of the credit 
for these improvements. 

The breakfast club originated in 1995, when 
Adams School began hosting the monthly meet- 
ings to create professional community and provide 
an opportunity for teachers to review research on 
best instructional practices. Although several 
rs at the school already kept abreast of cur- 




rent developments in the field, the school's profes- 
sional development efforts rested largely on out- 
side expertise, and they were too intermittent and 
variable in quality to have a long-lasting impact on 
student achievement scores. 

In interviews with the school's administrative 
team, Halverson learned that: 

► Faculty members did not want the breakfast 
club to be mandatory,' 

► The substance of the discussions needed to sell 
the program,' 

► The club should meet in the morning, so that 
teachers would be fresh and ready to entertain 
new ideas,' 

► The assigned readings should be kept short; 

► Teachers should be permitted to select the 
readings and lead the discussions,' and 

► The readings should align with the school's 
instructional priorities and the teachers' class- 
room practice. 

Innovation becomes institutionalized 

The breakfast club was designed to acquaint teach- 
ers with relevant research in reading and writing, 
to help them "work smarter, not harder," in their 
efforts to help students improve their reading and 
writing skills. A persistent underlying goal was to 
improve student test scores in language arts on dis- 
trict standardized tests. 

The Adams School leadership team recognized 
that improvement of student test scores might not 
result from a traditional professional development 
program using external consultants. Principal 
Brenda Williams realized that long-term gains in 
student test scores would more likely come when 
teachers had the opportunity to talk with one 
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Breakfast 

continued from pa^e 7 

another about their teaching and that 
the monthly breakfast club would stim- 
ulate such conversations. 

After two years, attendance at the 
breakfast club averaged about 75% of 
the school's teachers. The program was 
modified over time to add incentives for 
teachers to participate. For example, the 
principal encouraged teachers to lead a 
breakfast club discussion. Attendance 
increased as the veteran faculty mem- 
bers realized that they would be asked 
to lead discussions and therefore needed 
to find out what the breakfast club was 
about. 

More than six years later, the break- 
fast club has become an institution at 
Adams School, and over the past four 
years, student achievement scores have 
risen 22%. Teachers and administrators 
credit the breakfast club as a key ele- 
ment in creating the kind of profes- 
sional community necessary to develop 
a programmatic, cross-grade level 
approach to teaching reading and writ- 
ing in the school. 

The breakfast club resulted in signif- 
icant change for Adams staff. It pro- 
vided an opportunity for a school-wide 
professional community around lan- 
guage arts instruction. This professional 



community, in turn, was credited for stu- 
dent test score gains in reading and writ- 
ing. The club also helped make teachers 
take ownership of their professional 
development. And the documentation of 
breakfast club practice has given Adams 
School leaders an opportunity to reflect 
on their practice, discern patterns, and 
make sense of instructional initiatives 
that originally evolved in practice. 

Through a retelling of the breakfast 
club story, Halverson and colleagues 
have identified several of the guiding 
principles of leadership practice at 
Adams School, including: 

► The importance of patience while 
waiting for a voluntary program to 
take hold; 

► A commitment to considering 
research that is directly relevant to 
teacher practice, and 

► A continued willingness to use col- 
laborative design as a method for 
solving emergent school problems. 

Halversons current research is sponsored by the 
DeWitt Wallace-Reader's Digest Fund. With 
UW— Madison Education Professor Colleen 
Capper, Halverson is documenting and commu- 
nicating leadership practices for students tvho 
traditionally struggle. For more information, 
contact Halverson at halverson@education. 
ivisc.edu. 
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Measuring 
the content 
of instruction 

T eachers get lots of 
advice and support 
from a variety of 
sources about what to teach. 

But do they really teach 
what is described in content 
standards? Do they teach 
what is in the textbook? Do 
they teach what is tested? 

Classroom teachers are 
the ultimate arbiters of what 
is taught, and how. Regard- 
less of what a state policy 
requires or what a district 
curriculum spells out, the 
classroom teacher ultimately 
decides how much time to 
allocate to particular school 
subjects, what topics to 
cover, when and in what 

order, to what standards of achievement, and to which students. Collectively, teachers 
decisions, and their implementation, define the content of instruction. 

Knowing what teachers actually teach is important to educators and policymakers 
who need to determine whether and to what degree there is overlap (or alignment) 
between what is taught, what is tested, and what national, state, and local content 
standards prescribe. 

Tools for measuring content and alignment 

For the past 25 years, Andrew Porter and his colleagues have studied teachers' content 
decision-making in mathematics and science. F^e has developed three kinds of tools 
for measuring content and alignment: 

1 . Teacher surveys describing the content of instruction; 

2. Content analyses of instructional materials, including assessments,* and 
3. Indices of alignment between instructional content, instructional materials, 
a d standards. ^ 

FRir 27 
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Uniform content language allows one to compare alignment between 
states and to national standards. 
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Academic 
'enablers' critical 
to success 

Putting case 
study results 
into context 

Developing 
algebraic rea- 
soning in ele- 
mentary grades 
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FROM THE DIRECTOR 

Stimulating and measuring learning 

For many years researchers have studied the devel- 
opment of children's algebraic reasoning and the 
nature of classroom interactions that support that 
development. Tom Carpenter and colleagues have 
found that, if young children are given the oppor- 
tunity, they can make their own meaningful 
insights into the underlying structure and proper- 
ties of arithmetic operations. They can also con- 
struct ways of representing them and justifying them. 

Just as academic skills are crucial to students' success, so are acade- 
mic enablers. Academic enablers include motivation, interpersonal 
skills, and engagement. Steve Elliott and Jim DiPerna maintain that 
these enabling attitudes should be taught explicitly to optimize stu- 
dents' learning. 

Teachers are learners, too. Teacher professional change occurs 
through partnerships between university researchers and mathematics 
and science teachers. Adam Gamoran and colleagues recently explored 
how schools encouraged teacher collaboration through a combination 
of two elements: organic management and distributed leadership. 

And finally, this issue contains an introduction to a measurement 
tool I developed with colleagues that can help educators develop con- 
tent standards. The content matrix' aims to help educators make diffi- 
cult choices about curriculum, instruction, and assessment. Policymak- 
ers can use the tool to build standards that clearly define what is to be 
taught and what is not to be taught. 

For more information visit the WCER web site at www.wcer.wisc.edu 

Andy Porter 





The power of these tools lies in the uniform lan- 
guage they use for describing content. It is this 
uniform language that makes it possible to build 
indices of alignment. 

The language developed by Porter and col- 
leagues consists of uniform descriptors of topics 
(level of coverage) and student expectations (categories 
of cognitive demand). The level of coverage and the 
categories of cognitive demand form the columns 
and rows of a two-dimensional matrix. The con- 
tent of instruction is described at the intersection 
between topic and cognitive demand (See Table 1, 
Content Matrix] . 

The values placed in each cell of the matrix 
reflect data gathered from teacher surveys or from 
content analyses of instructional materials. For 
example, the surveys ask teachers to indicate, for 
the past school year, 

1 . the amount of time they devoted to each topic 

(the level of coverage), and 



2. for each topic, the relative emphasis they gave 
to each student expectation (category of cogni- 
tive demand). 

Porter and colleagues then analyze the results from 
surveys and content analyses to produce topo- 
graphical maps that graphically display the content 
of instruction. 

Porter's content matrix can also be used as a 
tool for developing content standards. The power 
of the content matrix in this context is that it facili- 
tates clarity in making hard choices. Porter says. 
State content standards — and even national-level 
standards, like the mathematics standards devel- 
oped by the National Council of Teachers of Math- 
ematics (NCTM) — tend to include much more 
content than can be taught in depth. Educators and 
policymakers can use the content matrix (and the 
topographical maps displaying the content) to 
build standards that clearly define not only what is 
to be taught but also what is not to be taught. 

Porter's method of measuring the content of 
instruction and alignment differs from other 
approaches in two ways: 

1. The tools allow independent and replicable 
descriptions of the content of instructional 
practice and instructional materials. The uni- 
form language for measuring content ensures 
descriptions at a consistent level of depth and 
specificity. 

2. The uniform language allows alignment to be 
measured across a large number of instructional 
materials and instructional practices. See Fig. 1, 
Vertical and horizontal alignment. 

Using the tools to study alignment 

Most approaches to alignment of assessments with 
standards start with a particular state's standards 
and ask: To what extent does the content in those 
standards appear on the test? Such analyses are 
unique to each state. They don't allow comparisons 
between states or comparisons between state and 
other professional standards. But the uniform con- 
tent language developed by Porter and colleagues 
allows one to compare alignment between states, 
and to national standards like the NCTM stan- 
dards. 

In a study done at the American Institutes for 
Research, Rebecca Herman and Laura Desimone 
recently used Porter's tools to study the alignment 
of standards with assessments in four states. Their 
data showed that the assessment of each state was 
no more aligned to its own standards than to the 
standards of the other states or to those of NCTM. 

Porter says that perhaps the state standards are 
not sufficiently specific to allow an assessment to 
be tightly aligned with them. But he says a more 
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likely possibility is that states have much more 
work to do to bring their assessments into align- 
ment with their standards. This finding is one 
about which U.S. Department of Education offi- 
cials are expressing concern, Porter says. 

The tools described here are used for science as 
well as mathematics. Some early work has also 
been done in the areas of reading and language arts 
and social studies. Following are examples of how 
the tools can be used to describe instructional 
practices, instructional materials, and alignment. 

Describing instructional practices 

Good measures of the content of instruction can 
serve 

1. to define the process of teacher decision mak- 
ing in reaction to the various messages that 
teachers receive about what should be taught, 

2. to describe the implemented curriculum or to 
measure the degree of implementation of a new 
curriculum, 

3. to validate transcript studies, and 

4. to provide the basis for powerful professional 
development experiences. 

With regard to professional development, Andrew 
Porter and colleagues are using their measures of 
the content of instruction as the core of a new pro- 
gram on teachers' instructional practices, with 
funding from the National Science Foundation. 
The program begins by having teachers complete 
surveys describing the content of their instruction. 
Porter and colleagues then analyze the results and 
produce graphic displays that are returned to the 
teachers. Finally, teacher teams use the data to 
answer the following questions: 

1. Is the content of our instruction what we want 
it to be? Is it aligned with our tests and content 
standards? 
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2. Are the differences in what teachers are teach- 
ing appropriate? 

3. Do the prerequisite courses provide the con- 
tent needed for effective grade-to-grade articu- 
lation? 

In research, indices of alignment between the con- 
tent of instruction and a student achievement test 
can be used as a control variable in studies of the 
effects of pedagogical practices on student 
achievement gains. An index of alignment can also 
be used as a descriptive variable in assessing the 
coherence of a state's or district's curriculum policy 
system. 

For more information see Porter's page at the 
WCER web site, www.wcer.wisc.edu. 




Figure I . Vertical and horizontal alignment 
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"Academic enablers" critical to student success 



A cademic skills are, and should be, the pri- 
mary focus of instruction in schools. How- 
ever, recent research suggests that student 
achievement also depends on academic enablers. Aca- 
demic enablers are attitudes and behaviors that 
allow a student to participate in, and ultimately 
benefit from, academic instruction in the class- 
room. These enablers include motivation, interper- 
sonal skills, engagement, and study skills. 

UW-Madison educational psychology profes- 
sor Stephen N. Elliott and WCER researcher James 
DiPerna maintain that enabling skills and attitudes 
can, and should, be taught explicitly to optimize 
students' learning. Their recent research finds that 
students' prior achievement and interpersonal skills 
influence motivation, which, in turn, influences 
study skills and engagement to promote achieve- 
ment. 

Study skills begin to assume a significant role in 
promoting achievement as students advance 
through the elementary school curriculum. Begin- 
ning in the intermediate grade levels, there is a 
shift in curricular emphasis from learning to read 
to reading to learn. The curriculum increasingly 
emphasizes content acquisition over skill develop- 
ment. Thus study skills assume a more significant 
role in the learning process. 

In a related study, Elliott and former student 
Christine Malecki (now a faculty member at 
Northern Illinois University) determined that 
interpersonal skills are a significant predictor of 
academic competence (the skills, attitudes, and 
behaviors that contribute to success),- and acade- 
mic competence, in turn, is a significant predictor 
of achievement. Elliott and Malecki concluded that 
social skills have a significant predictive relation- 
ship with academic outcomes. 

There is more evidence to support the relation- 
ships between academic achievement and students' 
competence. This evidence comes from research 
using DiPerna and Elliott's Academic Competence 
Evaluation Scales (ACES). DiPerna and Elliott 
have found that academic enablers measured by 
ACES affect student grades and performance on 
standardized tests of achievement. 

There are practical reasons for measuring acad- 
emic enablers. School psychologists and other 
education professionals need a framework for 
thinking about assessment, intervention, and pre- 



vention services so they can help students receive 
optimal benefit from their education. Elliott points 
out that failing to address academic enablers may 
result in assessment and intervention plans that 
overlook key factors contributing to a student's 
academic difficulty. 

Prior achievement is a strong predictor of cur- 
rent achievement (knowledge and skills). Likewise, 
current achievement is a strong predictor of future 
achievement. For students experiencing academic 
difficulty, chances for future academic success may 
be limited unless an educator intervenes to address 
specific problems. Delaying intervention to allow a 
student's skills to mature and possibly catch up to 
grade-level expectations may not be a wise choice, 
even for students at the primary level. 

The DiPerna and Elliott study suggests that 
students' motivation, engagement, study skills, and 
interpersonal skills should be considered when 
designing assessments for students experiencing 
academic difficulty. For example, a practitioner 
who designs an assessment focusing exclusively on 
motivation and current academic skills may be 
overlooking important things that contribute to 
the student's academic performance (e.g., study 
skills, interpersonal skills). This omission could 
result in identifying the wrong cause of the acade- 
mic difficulty. The educator also may develop an 
intervention that fails to address the true problem 
(e.g., difficulty getting along with others in class, 
which decreases a student's motivation to succeed 
in the classroom). 

Since academic enablers contribute in mean- 
ingful ways to academic achievement, and the pri- 
mary responsibility of schools and education pro- 
fessionals is to promote achievement, schools and 
educators need to consider what is being done to 
promote the development of academic enablers for 
all students. 

For more information, contact DiPerna at 
jdiperna@facstaff.wisc.edu or Elliott at 
snelliot@facstaff.wisc.edu. . 

Research funding was provided by the North- 
east Foundation for Children and the Fitchburg 
(Mass.) Public Schools. 

Information for this article originally appeared 
in School Psychology Review, 2002 , vol. 31 , issue 3 , 
pp. 298 - 312 . 
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Putting case study results into context 



R esearchers studying schools as organizations 
often confront a tension between the need to 
achieve an in-depth understanding of local 
organizational conditions, on the one hand, and 
the need to know whether the knowledge gained 
from such case studies can be generalized more 
broadly, on the other. 

Drawing on a national database helped WCER 
researchers Adam Camoran and Tona Williams say 
more about how a schools organizational context can 
support teachers' efforts to improve their teaching. 

Camoran and Williams and their colleagues 
were concerned with how schools and school dis- 
tricts support teachers' efforts to improve their 
teaching. In their examination of cases of teacher 
change in Wisconsin and Massachusetts, they dis- 
covered that leadership and autonomy were impor- 
tant aspects of schools' organizational contexts 
that supported change. The question then became: 
Was the level of autonomy and leadership they 
found in these cases really significant, in a national 
context? 

In two districts, Camoran and Williams and 
their research team interviewed teachers, surveyed 
them, and observed them in professional develop- 
ment seminars. In these districts, teacher change 
occurred through partnerships between university 
researchers and mathematics and science teachers 
as they developed classroom practices of "teaching 
for understanding" — that is, attending to student 
thinking, focusing on powerful scientific and math- 
ematical ideas and practices, and developing equi- 
table classroom learning communities. Camoran 
and Williams' study explored how the organiza- 
tional context of each site affected collaboration to 
teach for understanding. 

Camoran and Williams found that the sites 
encouraged collaboration through a combination 
of two elements: organic management and distrib- 
uted leadership. Organic management means that 
leaders respond to needs that emerge from teach- 
ing, instead of simply allocating resources in a 
bureaucratic manner. Distributed leadership takes 
advantage of expertise at all organizational levels 
and allows both teachers and administrators to 
make important decisions, instead of centralizing 
authority. These two practices enabled the schools 
to adapt to changes in teachers' thinking and class- 
room activities in ways that would not have been 
possible had the schools relied solely on more tra- 
ditional school organization. 



The study also found that administrators who 
adopted a style of organic management that 
responded to teacher initiatives faced two impor- 
tant tradeoffs: (a) when teachers were more 
autonomous, it was more difficult to establish a 
coherent direction for the school,- and (b) when 
teachers provided their own leadership, it was 
more challenging to ensure the completion of rou- 
tine administrative tasks. Across the sites, adminis- 
trators adopted a range of strategies to address 
these tradeoffs. 

In the cases studied, leadership in support of 
teacher change tended to focus either on establish- 
ing a compelling district vision that supported 
teaching for understanding or on providing teach- 
ers with the autonomy to develop their own 
visions. Each approach appeared to be effective. 
Camoran and Williams also found that leaders 
most effectively supported change when they dis- 
tributed authority beyond conventional leadership 
positions, while also finding ways to manage the 
necessary logistical details of professional develop- 
ment and other teacher improvement processes. 

Comparison 
to national averages 

Analysis of only two districts could not give 
Camoran and Williams the perspective necessary 
to interpret the broader significance of their find- 
ings. Without knowledge of national averages for 
the factors they studied, Camoran and Williams 
would have been limited to comparing the cases to 
one another and guessing about their overall 
importance. Therefore, when designing their sur- 
vey, they drew most of the questions from the 
Schools and Staffing Survey (SASS) of the 
National Center for Education Statistics (NCES). 
This allowed them to compare their findings with 
national data. When applied systematically, such 
national-sample data sets can bridge macro-/micro- 
level, and qualitative/quantitative, gaps in educa- 
tion research. 

In the case studies, Camoran and Williams and 
their colleagues found that leaders in the Wisconsin 
site emphasized providing teachers the autonomy 
to establish their own visions, whereas leaders in 
the Massachusetts site developed a district-wide 
vision to overcome a prior lack of coherence among 
teachers and schools. In survey responses, com- 
pared to national averages, teachers in both states 

continued on pa^e 8 
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Developing algebraic reasoning 
in the elementary school 




Effective teachers find 
instructional content 
that makes students' 
implicit knowledge 
explicit. 




C hildren in elementary school mathematics 
classes are often able to do more than the 
current curriculum challenges them to do. 
For example, they can learn to generalize and to 
express their generalizations accurately using nat- 
ural language and symbols. When they are given 
the opportunity, they learn to adapt their thinking 
about arithmetic so that it provides a stronger 
foundation for making the often difficult transition 
to learning algebra. 

School mathematics curricula in the U.S. have 
traditionally separated arithmetic and algebra. This 
historic separation has deprived students of power- 
ful schemes for thinking about mathematics in the 
early grades, says UW-Madison education profes- 
sor Thomas Carpenter. Carpenter directs the 
National Center for Improving Student Learning 
and Achievement in Mathematics and Science 
(NCISLA). 

Separating arithmetic and algebra makes it 
more difficult for students to learn algebra in the 
later grades. But simply pushing the current high 
school algebra curriculum down into the elemen- 
tary school won't work. A broader conception of 
algebra emphasizes the development of algebraic 
thinking, rather than just skilled use of algebraic 
procedures. Students in the elementary grades can 
begin to engage in meaningful discussion about 
mathematical proof and make significant progress 
in understanding its nature and importance. Devel- 
opment of their algebraic reasoning is reflected in 
their ability to generate, represent, and Justify gen- 
eralizations about fundamental properties of arith- 
metic. 






In their research, NCISLA researchers Tom 
Carpenter, Linda Levi, Patricia Berman, Jae-Meen 
Baek, Julie Koehler, and Margaret Pligge have 
found that when students working with mathemat- 
ics make generalizations and represent them for 
their classmates, they articulate unifying ideas that 
make important mathematical relationships 
explicit. Underlying this pedagogical approach is a 
conception of mathematical understanding as con- 
structing mathematical relationships and reflecting 
on and articulating those relationships. 

For the last 5 years. Carpenter and his col- 
leagues have worked intensively with a group of 
teachers to study the development of students' 
algebraic reasoning in the elementary grades and 
to construct instructional approaches that support 
that development. Their work with 100 elementary 
school teachers and their students in Grades 1 
through 6, including in-depth studies of three 
classes, has provided the following insights. 

A window on student thinking 

When students make generalizations about proper- 
ties of numbers or operations, they make explicit 
their mathematical thinking. Generalizations pro- 
vide the class with fundamental mathematical 
propositions for examination, while also opening 
up students' thinking for analysis and discussion. 
Although students often have a great deal of 
implicit knowledge of properties of arithmetic 
operations, they typically have not explicitly 
examined generalizations about properties of num- 
bers and operations or thought systematically 
about them. The trick for educators, says Carpen- 
ter, is to find an instructional context in which stu- 
dents' implicit knowledge can be made explicit. 
Discussion of appropriately selected true and false 
number sentences provides such a context. 

As an example: In one class exercise, children 
were asked whether it is true that 0 -i- 5,869 = 
5,869. After some discussion, the group came up 
with the generalization: "Zero added with another 
number equals that number." They also came up 
with the following generalizations: "Zero sub- 
tracted from another number equals that number," 
and "Any number minus the same number equals 
zero." In addition, one student came up with sev- 
eral related generalizations about multiplication. 

True, false, and open number sentences pro- 
vided a context in which these students could 
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A teacher records students' generalizations. 



begin to convert their 
implicit understandings into 
explicit generalizations. 

Number sentences generated 
by the teacher provided the 
initial basis for drawing out 
generalizations. But once the 
classes started to talk about 
generalizations, making gen- 
eralizations became a class 
norm, and students would 
propose generalizations on 
their own. 

In most of the classes 
studied, students would write 
generalizations on sheets of 
paper and post them in some 
location in the room. When 
generalizations were difficult 
to state clearly in natural lan- 
guage, the students would 
use symbols to express the 
generalizations precisely. For 
example, students repre- 
sented a conjecture about changing the order of 
numbers in addition as follows: For all numbers a 
and b, a -H b = b -H a. 

Condusions 

Elementary school students can learn to adapt their 
thinking about arithmetic so that it is more algebraic 
in nature. They can learn that the equal sign repre- 
sents a relation, not a sign to carry out a calculation. 
They can learn to generalize and to express their 
generalizations accurately using natural language 
and symbols. Although not all students in the ele- 
mentary grades will master mathematical proof, 
they can begin to engage in meaningful discussion 
about proof and make significant progress in under- 
standing its nature and importance. 

Understanding justification and proof takes 
years to develop. Although many sixth-grade stu- 
dents in a NCISLA case study were not yet able to 
generate proofs by themselves, most of them 
learned to recognize the limits of examples and the 
value of general arguments. They engaged in discus- 
sions of the nature of proof that made explicit 
important issues that most students never encounter 
at any point in their education. These experiences 
could provide a foundation for deepening these stu- 
dents' understanding of proof in the future. 

"One of the things that was striking about the 
classes we worked in was that the students were 
engaged in sense making," says Linda Levi. 'They 
thought that mathematics should make sense and 
that they could make sense of it. Students persisted 
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for extended periods of time working on a prob- 
lem, because they thought they should be able to 
figure it out." 

All students benefit by engaging in the kinds of 
interactions that are required to make generaliza- 
tions explicit, represent them accurately with nat- 
ural language and symbols, and demonstrate that 
they are valid for all numbers. Learning to use pre- 
cise language and communicate about mathemati- 
cal ideas addresses not only an important goal of 
the mathematics curriculum, but also important 
issues of equity. The best students have always fig- 
ured out how to derive generalizations and thereby 
make mathematics easier to learn and apply. F^elp- 
ing students make generalizations explicit gives alt 
students access to powerful ideas of mathematics. 

A new book is available. Thinking Mathematically: 
Integrating Arithmetic & Algebra in Elementary School, by 
Carpenter, Levi, and Megan Loef Franke (F^eine- 
mann, 2003). 

For more information about research into 
mathematics education, see the NCISLA Web site 
at http://www.wcer.wisc.edu/NClSLA/. 

Funding for NCISLA projects is provided by the 
National Science Foundation and the U.S. Department 
of Education's Institute of Education Sciences (formerly 
the Office of Educational Research and Improvement). 

Carpenter's study of algebraic reasoning in the ele- 
mentary grades is funded by the National Science Foun- 
dation. 
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Results 

continued from page 5 

reported higher levels of classroom 
autonomy and influence over school pol- 
icy, which suggests the presence of dis- 
tributed leadership. Teachers in the Wis- 
consin district reported a great deal of 
influence over the selection of instruc- 
tional materials and teaching techniques, 
and an extraordinary level of influence 
over school policy. Massachusetts teach- 
ers, in contrast, scored lower on auton- 
omy and influence. This pattern was 
consistent with interview responses from 
teachers, principals, and district staff that 
indicated the prominence of distributed 
leadership in the Wisconsin site, in par- 
ticular. 

At the same time, the Wisconsin and 
Massachusetts teachers both scored below 
national norms in the degree to which 
they perceived administrative personnel 
(and especially principals) as offering 
strong leadership and support. These 
findings, which suggest a lack of focus 
on centralized, top-down leadership, are 
consistent with Gamoran and Williams' 
interpretation that opening opportuni- 
ties for teacher autonomy means reduc- 
ing principals' vision-setting role. 
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Though Gamoran and Williams' 
study centered on developing an in- 
depth understanding of a small number 
of cases, the process of comparing some 
of their data against national norms 
made their analysis more robust. 

For more information, contact 
Gamoran at gamoran@ssc.wisc.edu or 
Williams at twilliam@ssc.wisc.edu. 

Funding for this research was provided 
by the National Genter for Improving 
Student Learning and Achievement in 
Mathematics and Science, supported by 
funds from the U.S. Department of Edu- 
cation, Office of Educational Research 
and Improvement (Grant No. 
R305A60007). Findings and conclusions 
are those of the authors and do not nec- 
essarily reflect the views of the support- 
ing agencies. 
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